Deploy and Use Open Source GPT Models for RAG

You should see the Open WebUI landing page.

You can hide the navigation panel by clicking the hamburger icon in the top-right corner of the panel. The New Chat button opens a new chat interface on the right side of the application using the default model, which is currently the Arena Model. The Workspace button opens additional settings where you can set the inference GPT model, upload files for RAG, create prompt suggestions, and even include tools such as web search. The Search option lets you search by tags. Currently there are no tags. Previous chats are located below the All chats drop-down list. Since you have no chat history the drop-down menu is grayed out, and there are no chats visible.

Notice the suggested prompts below the main chat window in the middle of the panel. Note that the default suggested prompts are generated every time you refresh or load the interface. Because this is a simulation, you may see the same suggestions. However, note that these suggestions might differ from the figures in the lab guide due to refreshing of the browser during simulation development.

In the top-left corner of the chat interface you see the currently selected chat model, which is currently the Arena Model.

Do not confuse the chat model with the underlaying GPT inference model. The Arena Model is just a placeholder for a fully-configured chat interface. You must select an inference model for the chat interface to be active.

You should see the models listed in a name:tag format followed by the number of parameters that make up the model.

The phi3, and llama3.1 are categorized as instruct models. Instruct models are specifically trained to excel at tasks that require following user instructions. These models are particularly well suited for chatbots because they can interpret to a wide range of user queries and commands and respond to them effectively. The mxbai-embed-large model is trained specifically for embedding tasks. This model is used in RAG applications for semantic search.

The models featured in this selection are classified as small LLMs because they possess fewer parameters. Models with a greater number of parameters typically demonstrate enhanced capabilities, handling more complex tasks and offering more nuanced responses. Each model here is designed to operate on a consumer-grade GPU with 8 GB of RAM, making them accessible for smaller-scale applications.

To estimate the RAM consumption of a GPT model, you can use the formula: M = P x Q/8 x 1.2, where M is the estimated memory in Gigabytes, P is the number of parameters in billions, Q is the quantization in bits, and 1.2 accounts for additional overhead related to inference optimization tasks. Quantization defines how many bits are used for storing each parameter in memory. It's important to note that while quantization helps reduce memory requirements by compressing model size, it can also lead to a decrease in performance.

For instance, the phi3 model, which has 3.8 billion parameters and uses 4-bit quantization, would have an estimated GPU consumption calculated as follows: 3.8 x (4/8) x 1.2 = 2.28 GB of RAM. Similarly, the llama3.1 model, with 8 billion parameters and 4-bit quantization, would require approximately 4.8 GB of RAM. This approach ensures that the models are both efficient and effective within the constraints of typical consumer hardware. To put things into perspective, GPT-3 has 175 B parameters and runs 16-bit quantization which consumes around 420 GB of RAM.

The RAM consumption for RAG applications can be calculated by considering both the inference and the embedding models. For example, the mxbai-embed-large embedding model with 334 million (0.334 billion) parameters and using 16-bit quantization, requires approximately 0.8 GB of RAM. When combined with the llama3.1 model, the entire RAG system has a baseline RAM requirement of at least 5.6 GB. Additionally, optimization processes such as data caching, activation map storage, and parallel processing overhead are actively running on the GPU, all of which demand extra RAM. These processes are essential for efficient data retrieval, quick access to frequently used data, and managing the computational load across multiple GPU cores. Therefore, allocating 8 GB of GPU RAM for the system provides sufficient headroom to accommodate these demands and ensures smooth operation under varying workloads.

Notice how the chat model name changes to the inference model name.

You should see the chat interface change slightly.

In a real environment, this transition happens in real time when you click and start writing the prompt.

In a real environment, you would first see only your text, and after a few seconds, you would see a completion suggestion from the system.

You should see an example of automated prompt completion.

Similar to how prompt suggestions are randomly generated with each page refresh, the prompt completions will be different even for the cases where you use the exact same starting prompt. This is due to the non-idempotent nature of GPT models, that are used to generate these completions. The non-idempotent nature means that using the same input does not necessarily produce the same output.

Since this is a simulation, you will get the answer instantly.

However, in a real environment you would normally see a loading screen first.

In case of the very first query, it takes a while for the system to load the GPT model onto the GPU. Specifically, the llama3.1 model, which contains approximately 4.9 GB of data, needs to be transferred to the GPU memory. This process, which is known as cold-loading, can be time-consuming but is essential for preparing the model for efficient operation. Cold-loading occurs when the model is loaded from scratch, which takes a couple of seconds for smaller models like llama3.1. However, imagine loading a much larger GPT-3 model that encompasses around 350 GB of data. Once loaded, any subsequent usage can benefit from what is known as hot-loading, where the model remains in memory, ready for immediate use without the initial delay.

GPT models usually stream the answer as it generates in real-time, which ensures that the AI system feels responsive by matching the speed at which an average person reads silently—typically 200 to 300 words per minute, or about 3-5 words per second. Thanks to the parallel processing capabilities of GPUs, such system can deliver outputs even faster. However, it's important to note that models with higher parameter counts might require more time to process the same input, potentially affecting the response speed.

Since this is a simulation, you will only be able to choose the regenerate option.

The options as they appear from left to right are as follows:

• Edit (pencil icon) allows you to change the response and copy the final result.

• Copy (clipboard icon) allows you to copy the response as is.

• Read Aloud (speaker icon) uses the text-to-voice capabilities of the app and reads the text to you, provided you have sound properly configured on your computer.

• Generation info (the information icon) shows performance metrics of the response, such as response tokens per second, prompt tokens per second or the total time it took for the entire query. You can hover your mouse over the icon to see the results for your query. Note, token count is not the same as word count. Word count can be estimated as 3/4 of the token count. For example, 100 tokens per second is roughly 75 words per second.

• Thumbs up and down icons let you evaluate the answer as good or bad to help you keep track of the chat history and qualitatively assess your interactions.

• Continue Response (the play icon) instructs the model to continue with the generation in case the answer streaming pauses, which might happen when the answers become really long.

• Regenerate (recycle icon) lets you process the very same prompt again. This is helpful when you want to take advantage of the non-idempotent nature of GPTs and get a slightly different answer to the same query without copy-pasting the same query again.

It is the last option on the right.

You will see a different answer compared to the one before.

Notice now a <2/2> option appearing to the left of the pencil icon. This lets you know that you are currently viewing the second out of two possible answers to the same prompt. In a real environment, you can click the < symbol to go back to the previous answer, whereas clicking the > symbol will move to the next one.

The chat names are dynamically created. In this case, your chat is filed under the name Laughter is Contagious.

Keep in mind that in a real environment, these chat classifications are dynamically generated and the exact same chat might get filed under a different classification.

You should see several options pertaining to this chat.

In a real environment, you could select any one of these options, but in this simulation you will only be able to choose the Delete option.

You should see a pop-up window asking for confirmation.

You should now see a clean workspace without the previous chat.

Now that you understand the basics of the Open WebUI, you decide to configure it for RAG.

You should see a pop-up menu containing various settings.

The Settings option lets you configure general UI settings, such as interface layout, personalization, and so on. The Archived Chats option lets you access the saved chat history for the currently logged in user. Playground offers an easy-to-use chat interface where you can define and test various System Instructions, which are detailed guidelines for how you want your chatbot to behave and respond. The Admin Panel lets you manage GPT models (on-prem or cloud provided), and RAG settings, among others.

You should see the Users tab selected with the Overview view opened.

Notice how the currently selected Users tab is colored black, whereas the Evaluations, Functions, and Settings tabs are gray.

The Evaluations tab allows you to view how users of this Open WebUI instance have rated the models that they interacted with, based on the thumbs-up and thumbs-down scores for each chat. This feature is currently in beta in version 0.4.7 of the application.

The Functions tab enables you to discover, download, and explore additional custom tools for Open WebUI, including advanced web search features and custom Python code that can be integrated with GPT models.

In the Settings tab, you can find GPT model, RAG, and other advanced settings.

You should see the Settings page where the General tab is currently selected.

There are a lot of settings for a complex and versatile application such as Open WebUI. You will focus only on the settings that are important for the RAG application and leave the rest at their default values. The settings you are interested in can be found under the Connections, Models, and Documents tabs. You will explore and configure these settings in the following steps.

You should see the OpenAI API and Ollama API sections.

Here you can manage access to the on-prem or cloud-hosted inference servers that communicate over the OpenAI API.

Here you can enable or disable the OpenAI API connections whenever you need to. You can also add URLs that lead to the cloud-hosted (default) or on-prem servers by clicking the + icon.

If you want to use the cloud-hosted OpenAI inference server, also known as ChatGPT, you have to be a registered user with OpenAI and own an API key. You can add the API key in the corresponding text field next to the URL that will serve as the default API key. The cog icon in the same row lets you define which models from their portfolio you want to use or even use a different API key per model.

You will not be using cloud-hosted services, and you don't have access to any on-prem OpenAI inference servers.

You should see both the OpenAI API and Ollama API in a disabled state.

You should see the Ollama API connection enabled and set to the default settings.

Here you can manage the connections to the Ollama inference server instances. This simulation is based on a locally hosted Ollama inference server.

The default URL is set to localhost on port 11434. In a real environment, you could connect the OpenWeb UI to another Ollama inference server by configuring the URL and port of the other instance.

The cog icon lets you configure additional settings, such as adding a prefix to the existing connections to avoid conflicts. Because there is only one local connection, you can ignore this setting.

The wrench icon allows you to manage the embedding and inference models hosted by a specific Ollama inference server. Even though the models used in this lab guide were already downloaded, you will explore these settings in the following step to learn how to find and add other models to the system.

You should see the Manage Ollama pop-up window.

At the top, you should see the Pull a model from Ollamma.com section. In a real environment, you can choose to download any of the models hosted on the Ollama model repository by typing the models name and accompanying tag. You will explore how to get this information in the next step.

You can delete any of the downloaded models in the Delete a model section by choosing it from the drop-down menu and clicking the trash can icon. The next figure shows this example.

For more advanced use, you can also augment some of the run time model parameters of the supported models in the Create a model section. You will be using the default settings in this lab, so you can ignore this part.

Expanding the Experimental section reveals the support for downloading models in the GGUF format directly from the Hugging Face platform. This feature was still under development at the time of writing this tutorial and exploring its functionality is not included in this simulation.

In a real environment, you would see the Ollama model library open in another tab. In this simulation, you are presented with the full screen view of the tab.

Each model has its name written in bold, a description and a number of parameters. The library is frequently updated and you might see different models than the ones shown in the figure if you opened the page outside this simulation. Looking at the figure you see that llama3.3 has 70 billion parameters and, according to the description, offers similar performance compared to a much larger llama3.1 model that uses 405 billion parameters.

Additional model details such as quantization are available by clicking the model name in the library. You will explore llama3.1 variants, because this is the model currently configured in the OpenWeb UI.

You should see llama3.1 in the middle, among other models.

You should see the default model variant for llama3.1.

The table shows the last time that this variant was updated. At the time of writing this tutorial, this model was two weeks old.

The model row tells you the GPT architecture, number of parameters, and level of quantization. In this case, the architecture is llama, the model has 8.03 Billion parameters, and it uses 4 bit quantization, denoted by the Q4 prefix of the Q4_K_M string. The K_M demarcation refers to the quantization method and optimization.

The params and template fields are used to control the behavior and formatting of the model’s responses. For example, the stop sequence in the params field defines when the model should stop generating text. The template section defines how the input prompts are formatted before being passed to the model, which is already taken care of within the OpenWeb UI.

You should see the 8b, 70b and 405b parameter variants.

The drop-down shows three variants, which does not mean that there are only three available variants. The three variants shown in the list are the Q4_K_M quantization variants. The View all menu option reveals all variants.

You should see a list of all variants that are offered by the llama3.1 family.

If you want to download any of these specific models, be sure to use the full name written in bold. Look at the figure of model variants. To use the llama3.1 405 billion Instruct model with quantization of 16 bits, you would have to use 405b-instruct-fp16 as a tag. So, a full model name in this case would be llama3.1:405b-instruct-fp16.

To download this model, you would write llama3.1:405b-instruct-fp16 in the Pull a model from Ollama.com field and click the download button, as shown in the next example.

You will return to the Settings view.

You should see a list of all the currently downloaded models.

Here you could use the toggle buttons to effectively hide the models from other non-administrative users. Clicking the pencil icons lets you add detailed descriptions, default prompts, and other metadata that all the users see. Leave all these settings as is because you will configure your own chatbot using one of these models as a baseline after you are done configuring everything for RAG.

You should see all the settings for document processing and RAG.

You can download various embedding models and reference them here to use with RAG. The Embedding Model Engine is currently set to SentenceTransformers. Sentence Transfomers is a model repository, similar to Ollama's model library and Hugging Face portal. You will use the Ollama server for the embedding model engine.

You should see a drop-down menu with the Ollama and OpenAI options.

The OpenAI engine points to the cloud-hosted embedding model provided by OpenAI. Same as for the OpenAI inference model, you must have a registered account at OpenAI and an API key. The same pricing policy applies in this case as well.

You should see the http://localhost:11434 URL, which points to the Ollama inference server.

Below the Ollama URL, you should find the Embedding Batch Size parameter, which is currently set to 1. The embedding batch size determines how many text chunks are fed into the model at the same time during the embedding generation process. Larger batch sizes allow the system to process multiple text chunks in parallel, increasing throughput and reducing the total time required to generate embeddings for large datasets. Each batch of text chunks consumes memory that is proportional to the size of the text, the batch size, and the complexity of the model. Smaller batch sizes are more memory-efficient and suitable for devices with limited resources.

Next, you will see the Hybrid Search option, which is currently set to off. Turning this option ON enables the use of re-ranking. Re-ranking is a process that takes the initial list of retrieved search results based on their relevance scores. The re-ranking process then refines their order using a more advanced and specialized model called a re-ranking model. While the initial search focuses on quickly finding documents that match the query, re-ranking dives deeper into the context and meaning of both the query and the retrieved results. This ensures the most relevant and high-quality matches are prioritized at the top of the list. Re-ranking is particularly useful when you need highly precise results in scenarios where relevance can significantly impact user experience, such as question answering or document retrieval in a critical domain. However, re-ranking requires an additional specialized LLM that is trained specifically for this task, which adds computational overhead. You will not use re-ranking in this simulation.

Further down in the interface, you will see the Embedding Model pane.

Here you define the embedding model that you want to use. You can download models in the Connection settings, as described in the earlier steps. The mxbai-embed-large model was already downloaded for the simulation. You will set this model for your RAG application. There are a lot of different available embedding models and they differ by how fast and in which manner they convert text into vectors. Please note that changing the embedding model requires you to re-process all the text because different models create different vectors from the text. The mxbai-embed-large model offers a good compromise between the quality of the embeddings (text-to-vector translation) and processing speed.

Your screen should now look as follows:

The Content Extraction settings define the engine that you will use to extract the text from the documents and parse it. The default setting uses an internal mechanism for this process. In a real environment, you can click Default, which opens a drop-down menu where you can select the Apache Tika server. Apache Tika is an advanced parsing engine that can detect and extract metadata and text from over a thousand different file types such as PPT, XLS, and PDF.

You should see the Query Params section.

Notice that the Top K parameter is set to 5. This parameter specifies how many of the highest-ranked database entries are selected from the retrieval phase based on their relevance scores. For example, if Top K is set to 5, the system retrieves the 5 most relevant entries for a given query from the database to be used as context. The context is then passed on for further processing such as re-ranking or directly feeding into the generative model. If Top K is too low, there’s a risk of missing relevant information, leading to incomplete or inaccurate answers. On the other hand, if Top K is too high, the system may process irrelevant content, which could reduce the quality of the response.

Below the Top K parameter, you can see the default RAG Template. The default RAG Template acts as a system prompt specifically designed for queries that require retrieving information from the knowledge base (vector database). Without this template, the GPT model might generate irrelevant responses such as stating that it found information in the knowledge base. The template also includes explicit instructions for the model to acknowledge when it doesn't have sufficient information to answer a query. This guidance is crucial in mitigating hallucinations and minimizing inaccurate or erroneous responses. By clearly defining the model's behavior and providing a structured framework, the RAG template ensures more accurate and reliable outputs while maintaining alignment with the retrieved knowledge base. The structure of the system prompt is carefully crafted using XML-style tags, which play a vital role in distinguishing instructions from placeholders. These placeholders are used to insert the retrieved context and the original query dynamically. Note that you can use any markup language or templating syntax to define the placeholders for the context and the query. Do not change the template because it can significantly impact the responses of the RAG system.

You should see the Chunk Params pane followed by the rest of the Documents settings.

The Chunk Params section is used to configure the Chunk Size and Chunk Overlap parameters, which play a critical role in splitting text into smaller, manageable pieces for representation in a vector database. These parameters directly influence how text is segmented and stored as vectors, affecting both the accuracy of search results and the performance of the retrieval process.

The Chunk Size parameter determines the number of characters or tokens that will be included in each chunk of text. A larger chunk size allows each vector to represent a longer span of text, which provides more context and can be particularly useful when the goal is to retain broader ideas or generate summaries. However, larger chunks can make the search results less precise because they cover more content within a single vector. Conversely, a smaller chunk size divides the text into more granular pieces, which allows for finer search accuracy and improves the system’s ability to retrieve very specific details. For instance, a chunk size of 200 characters works well for highly detailed queries, whereas a chunk size of 1000 character is better suited for broader queries or long-form content where retaining larger contexts is critical. You can leave the settings for Chunk Size at 500 because this setting offers good granularity for the prompts that you will use during the simulation.

The Chunk Overlap parameter determines how much of the text in one chunk is repeated in the next. This overlap helps preserve continuity between the chunks, ensuring that important information spanning across boundaries is not lost. Higher overlap values are particularly useful when dealing with text that contains flowing ideas because they maintain context and coherence. However, they also introduce redundancy and increase the overall storage size. Lower overlap values, on the other hand, reduce this redundancy but may risk cutting off meaningful information at chunk boundaries. For example, if the chunk size is set to 500 characters with an overlap of 50 characters, the first chunk includes characters 1–500, and the second chunk begins at character 451, extending to 950. You can also leave this setting at 100 because this value proved to be sufficient considering the Chunk Size and the files you will be working with.

You should see green notifications informing you of successful configuration.

In a real environment, green notifications disappear on their own. Your screen should now look like before you clicked Save.

With the RAG configurations in place, you turn to the documentation regarding using your own files with RAG.

You should see the Workspace page.

The Workspace page is used to manage so-called Models, or ChatBots. You can define a ChatBot to use a specific inference model, add instructions on how you want the bot to behave, and even add a database for the model to use. For now, you will upload the output from various show commands and make them available for all ChatBots.

You should see the existing knowledge collections that were pre-loaded for this lab simulation.

A knowledge collection comprises one or more uploaded files that are stored in a vector database. The collection serves as a single reference to multiple files so you do not have to specify to GPT which file to check for data on each prompt.

The first collection on the left, titled Nexus 9K Complete Configuration Guide, includes the entire PDF of Nexus 9300 Interface Configuration Guide downloaded from: https://www.cisco.com/c/en/us/td/docs/dcn/nx-os/nexus9000/105x/configuration/interfaces/cisco-nexus-9000-series-nx-os-interfaces-configuration-guide-release-105x/m_overview_9x.html

The second collection, named Nexus 9K L3 Interfaces Guide, includes only a PDF of a chapter for Layer 3 interfaces from the entire configuration guide.

These two collections will be used to show how much influence splitting of the initial dataset has on the quality of a RAG system.

You should see the Create a knowledge base page.

The page should look similar to the following. Note that the page in the lab simulation might look a bit different due to the text input objects in the simulation.

You should see a green banner informing you of the successful creation of the knowledge base.

In a real environment, the green banners disappear after a couple of seconds on their own. You should now see only the Fabric Information collection page.

Here you can manage, add, and delete all the files associated with this collection.

You should see a drop-down menu.

The Upload files option lets you upload individual or multiple files at once. The Upload directory lets you define a directory from which all the files will be uploaded. The Sync directory option deletes all existing files and replaces them with new ones. The Add text content option lets you add the text that you write yourself.

You should see a file browser window open with all the files you intended to use for RAG.

Notice two types of files, PDF, and TXT. The two PDF files are the configuration guides mentioned in previous steps and are already part of their respective collections. You are interested in the TXT files containing the information about deployed Cisco Nexus 9300 switches.

The TXT files contain the output from the various show commands describing the fabric that you are currently working on for the company. The fabric contains one spine and two leaf Cisco Nexus 9300 switches in a CLOS topology. The fabric is configured as BGP with EVPN overlay and also includes all the necessary QoS configurations for AI/ML workloads, which is quite a complex configuration. The TXT files are named by the switch (leaf01, leaf02, spine01) and the command to obtain the output. Each file also has this information written in a header.

This kind of data preparation is necessary for RAG, since it relies on semantic similarities between the prompt and the data contained in the files. In other words, you can't expect from a RAG system to provide data regarding leaf02, if leaf02 is not mentioned in the text itself. For now, keep this in mind since you will put this to the test in the following steps.

Notice that only the text files (.txt suffix) are selected.

In a real environment, you would have to select the files yourself. However, placing these files in a directory and using the Upload directory option would make this less tedious.

You should see all the files processed and their names written on the page.

In a real environment, you see how each file is uploaded and converted into vectors. Each time the system is done with one file, it prints a success message, outputs the filename, and moves to the next one, as you can see in the figure.

You should now see three collections.

You can see when a collection was last changed in the collection card. Notice that it states that the Fabric Information collection was updated a few seconds ago. This update was triggered by the uploading of the TXT files.

The PDF files were uploaded using the same procedure. Although you have learned how to upload files using the output of show commands, this time you'll begin by exploring how the RAG system works with PDF documents. PDFs are easier to understand and provide a more straightforward approach for learning how RAG handles citations.

You should see a brand-new chat interface window with the default Arena Model selected.

You will change the model to llama3.1 and set it as default in the following steps because you will be experimenting and frequently opening new chats.